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*-*-y h©«fcdfc*'y h7— 5> 3 3 2 icjfttttfftTfi 
y, *«y h7-7 3 3 2fcx-*£2&fflL*'y f-7-7 
3 3 2fr5x-*£gt«J£o *>yh7-73> 



hP-5 3 2 6li3> hP-/U37> KtT.x-773 
?>KO«HR*fTd. T'Py'yA^T-' **-B3ttfclB 
If LTfc< ^P-fe-yy-3 2 1 Wfi©^5?a-;U 
jtM^EU 3 2-7««fflr*. *yHIXlB*a>hP-7 
3 2 Sttyt- hu> hP-;USB3 0 9fr53?V K 

*sw«y» *©a?> K*y« u 3 2 7icfBirr 
**f»ttB»^o*y*t3'2 1 ^aj-rs. 

[0 0 2 9] *-r-f*7'nty1t3 2 4btt/»3 3 

OKttfltttiTi/*. 7p-fey+>-3 2 1 ttav hP-/b 

y-y3 2 4blC)MajL.x *ty3 2 7» *-y K7-73 
> hP— 53 2 6fc%tH*/\- Kx-f X73 V hP— 5 

3 2 5fr64—7<f *7*P-fe'y+f3 2 4 btCftLTx- 
*Rtf3?VKtfiBSh*«l:3fcr*. *- xV*7p 
■fe-y-9-3 2 4 bttfr- 7/1/ 3 3 3 blCfcy*- x-f *7 

>73 2 4c MMtttiTfeUt *-?4*Ty?i 2 

4 c tt-r-7/U 3 3 3 c (c* y Xt?-*j 320a (C&R 
**i"cV*. *-xV*7'P-fe < y-y3 2 4 b»i*-x-f 
#7>73 2 4c tXtf-* 320a 

WSL «HO«WB#a5*^ttxUetta6t^-afc** 
ai7J*-B:5o *-x-f*7 , P-fe-yy3 2 4bi:*-x-f 
*7>73 2 4ctt, MttMc*- xV*7*o-feyt«: 

*-x<*7>7 p ojR^«a$-r«3>tfa-^-y7v 

Tt>«fclA, 

[0 0 3 0] fi-t3 2 3(ix + >^.;U2 a 6^6215 

nr* fc»2S^ 2 & e cf u e v a >ffl**»ir 5„ 

^'57 y 7 3 V h P— 5 3 2 4 a 7-r y 7 3T> 

•7 - 77-y73>hP-53 2 4attt; - 
x*'T-7;U3 3 3a ^LTtrx^x-rX7U>r 3 0 
2tCg^JtlTl^,, MPEG — 2x3—^3 2 2(i, 
/N-Kx-r^^ ^'5^73 0 IRtfxn-xS 2 36^5 
OS. m titc '.9*9- 9 Z WMct So 

[003 1] E4tt, y*-h3>hP-il/gg3 0 9 
©jEiBH?**. 'J*- h3> h □-;!/*■ 3 0 9lc« 
6-3©-ftry-5/3 fiP^ ±#^>3 1 0, 

fe^V3 1 1 > T^>3 1 2, &$*>3 1 3, 3 
S?*4r>3 1 4aift-K9-'T^/U**V3 1 5««t»b 

[0 0 3 2] :WcE|5«!:iI6lC^L/c7P-x1'-h5 
#»a L^6^5*HJfifl5Sllc J; £.->7sxZ*©!Simco^T 
IKB^-r^ EI5lC^Lfc7P-x + -h«P<x-f73> 
x>'y7P/U^1> )K2l^2SWl)PliiS7P/K^ 
4lcJ:oTll^*n?.Stta*mLfc i ta)T*S<, 06tc 
iLft7D-ff- H*«K#ffl->XxA3tCcfcoTII 

[0 0 3 3] S3 00tCfc^T, t'x^3>x> 

"X *-x-(-^-3>x> , >^t>^X'r73>x> , yAi < 

p<x-i'73>x> , y7p/W^iic < fcy^*n5. $ 
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»g"jr— fmzttiz. mmmiT-t. bps, 1 

Utr»>M:£xt?3537 K/t>* h • rU^'aV • ->X 
7Z*-=l = -yx-f (ATSC) TB7p^A<D§®J3iJK 

tt«tt*» A/ 5 7 teEB**lT^». 
[0 0 3 4] S 3 0 1 Kfebv?* ttftHt$B7D/tf #4 

-f VS-^x-f ?x > hfcttJBx— * <b LTftfiE 
f*. x5>*;Utt2«itt?J4» fcSao^x—Sffcg 

h • xUfcf^aV' ->7xA • 3;7f-f (ATSC) 

tf $6£ra£ffc-r5fctt<D«te£Si*&£» A/ 6 5 T£A& 

tf, 7 K/tVT. h • xUtfv'aV • x>/\>X*V h • 
7*- : 5L, (ATVEF) T'B, HTML 4. 0. EC 
M AT.? ' J 7 K&tfD OMlcSrK ttT(^%. 
DVB-MH PSlf A T S C - D A S E<DJ:5&fte<7>$ 
teB J A V A T M zfa a y ?S1§{cg-3 < fctflTfc 

[0 0 3 5] S30 1TB, ttjinltiST'P/^f * 

4B, ySx—frii-f'VS-^x-i'TXU^Vhx- 

* * ^ * r — 9 <t -r v * — 5 * x -< ?x u * > h &mm 

ZmZCttfTZZZolzLT^Zo WiZ. S30 1T- 

b, mmmyuiuvut**?— »£<(•>*-< ?<? 
< b< v * -5 * x < ?x > h flD^ft wflw»i5iat 

X > KD8tSf#ffl->Xx/» 3 7®MtftttaCT%ft 
^»5t><DTS5o ^x-^OS-^x-f? 

B&arr*. 

[0 0 3 6] &(CS3 0 2?»> fifc&i2B;><xV7x 
-*<htt«x-*££»b?*. ttJIx-$KB, S3 

o o t s 3 o i ftzmzmzmBmm® \ot, 

fi«MJi 2 B h 7 h 7. h 'J - ht LTOM P E G 

-2£ffl^T7i>'$/b»iM£*T5o ?-lfX7j|/x7*U 
*+J-/f«5>;*#-h->XxJU2 3 0B, ISO/IE 
C 1 381 8-6T<Tktb<s>tlT^%7 ! y*l\''7>f— 
[sv-J> *xV7- P^VKSlfpyhP-iU (DSM 
-CO ffl©BB?Sltt, iDPSffllcti^UjILr-^e 



[0 0 3 7] £fett2 4 2(C«fc >Jf6©x 

+ V:^l/T*£»bfI^££1B*tf> R F/i£fir>7.xA 
243<MTSC Sb^it D V B (Dcfc -5 frlf iSxv^/U 

[0 0 3 8] H60)S3 0 4lCfe^T. HB$#ffl->Xx 
^3T*Bfifc^2fr5gfmi^&g«-f£>o xa-x3 

2 3ii«§oa bps, m-ox+v^i/'Mi^KWf 

«^£ttHJf*o C©x^*/l/ft^ltB.><xV7x- 
^x-SKtHV^-^x-r^xi^y htf^ 

[0039] S305fcfcl/»T, MPEG-2xP— 9 

3 2 2«. m*<0T\i-ry , )'r-\ y*y >yu?^~y 
-^ict* y ^ t— ? •h'T 7x > h 

[0 0 4 0] S3 0 6?'l*> p<^ l J3 2 7rtlCx-'?';U 
<hLT, **^tty\- Kr-fX^ K7<^3 01 Icp- 

'Tv?— 5?x-f ?xu><> hro^n^n*7— j» • p 

Vt>7 • Xh'J-MCieifr^cfc^^PHl-y^S 2 1 
6WSaa^ff5o S 3 0 1 fcfcVT^r— JtStf-T 

WT^i5JB^r5S3 1 0-p«T*ft*x-r S"\© 
7 * * XflHlfcff ? SS<D* - <!: £ 3„ 
[0 04 1] S3 07lCfc^Ts yP-ty-9-3 2 IB, 
y 9 9— Z Xlf* y 9 — ? 9 x ■< 7x b * y h ft 5 X« 

? r — 9 £ 4 y 9 — y <? =r 4 7x U t< > 

HcfljSttttLTtsirr^o c:a)*i5iEiiW7jaB, *fis 

?i:-f>-S»-5?7 1 -i'7xUp<>h*, J-X 

Ttc^-rss3 1 oicfciNT^LK^as-rfctoron 

2©*-<!:LTffl^6ni>„ 

[0042] S308lCfe^T> hP 
—5 3 2 4£Zfa-tiv*t3 2 1 ft'iMf LT, S3057 

*7-fX7H'3 0 2±(C^7<'7P>x> > y<DS^ 

i"(3 0 2±izmmtz* : T'(73y : Tyv£LTs m 

> Ka— tfBbfx^x-f 3 0 2T-C^f-<7 

P>x>'y^«lS-ri>» Hgf^tChoT^x-T^PVT 1 

awtt-KWJ, j-xm^x^7P>-r>^*^-rs 
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[0 0 4 3] £(Da>-?>y^E-KB$lCfc^-e\ * 

*x— ? N -ot-vtTjyzi'tyb&zwtcn 
S/Txtt * x -r 7 a >x v> <085*±fc«fcr 

Mmt-tftzmmit. xm#&%s. 8 4 8. 352^ 
tmmmm&&z* zoxmrn^vms. 8 4 8, 35 

2 n*m*Z1CMt. LTfEIB LTfc < o 

[0 0 4 4] S3 09lCfc^T. 40©Stl*§iitfj#JI© 

3 0 96^s*n, *©A*i£gfrr£. bps. a*©s 

Mc7*-fe*tau:tt, «nHftta>TV*y«Wre-K 

#5u*/ui Rc->xxL>*«jyg 

«W»y4-l*avKn-;u«ll3 
0 9*fln/*Tc«W!jy*jL we. BrS©y*x-*<!: 

«»r***c coiat»»Ttt» #«© 

ttanra. s^xa*^-, nay>** zmzitm 

7>yilHLft^ * x~ * <t< VS-^tV 71 U 

lTStl7V3. 
[0 0 4 5] aVxWWR-KOUBs 'Jt-haV 
hO-jl/S«3 0 9©tf'£>1 10-11 3<0f-ftfft 

±#*v3 1 oic«3sacDS«aic^-r5i¥jaitfai©»a 
#«yaTShT*y» s#*>3i 3icttiiay>* 

ttttWBiOflltf. T#*V3 1 2fctt$/*xA*x 

a-©»SK«yyaT6tiT^*. awafxrit 
a yf-yyw&'E- K'#S i ©#st- K^y 
giisct^T'tSo «y»jt*fT3icfck mil©** 

[0046] S 3 1 0 fcfc^T, »R LfcXftttttftfll 
tt7*x-r7ft8SreSy»" j«RLfc«i5»W»atfa 

5*-*fc-fv*-5* x^xu^vhtftg^assti 

3. «mjr-*tt*xV7a>x:/*:>U:S»&*ft 
TMfl)?, h h ***** *3 2 0(c(i$&£R 
*©* x -f 7 a >x > yoWMWJx- 2 ©If $Btf£ 



^7avx> , yicffli&&y$x-$W>*:-57x 

* * x— * w >* - 5 f? t < 7*x y h tivwm- 
[0047] ii^a£tt/c*Sx-s<h-r>s-^x 

(SLrctfr*T-f^7"U-f 3 0 2±©ffl»S'7'f> K-7 

Tl^y xV 73>f>7 <»:«:*IH&©.><* x-*W 
5?x-< 7xU* V hf£W-££A,7£fc©TS3*B 

£ic«> »OT***vK9tt^**iT^**xV7 

[0 0 4 8] $tcs S3 1 OlCfctNTx «itf*>1 1 

o~i 1 3©«ii6tt'>XTi»fcj:ysKJ£*n3. bp 
*>. ggv Ms »t*ar4o©ita**>a:s a 

*>1 1 0~1 1 3(z«MEB£fn«. 

#*V1 1 0~1 1 3 tt. HSlCwtftiTf***^ 

gijr- *ic * y a* *3„ nwt* v& wa*« >©» 

[0049] §&©7sx * S 3 1 1 Wfc a- 

+f tfttEO'T V * - 5 * x f 7*x > MCJS UfcJfcT 
^5<7-^gL<(i'f>$-^f^7iL/y>hiJt 

[0 0 5 0] CCr, S 3 0 8~S 3 1 1 (Cfe^TUff 

*xV;*7U-f 3 0 2©4^©a<!:Ma^t5tlT^ 
a-/USS3 0 9±©9cEP/Jt^>t)%-r73lRl(c, 

Witt«Mii=«r*y ^x— > ir-r y*- jvt* ij. 

0 2 ©aj£f ssfrsrtfflijtejiijs^* ti«. 

[00 5 1] SfflPJiTJS 

ei 7 tc,Tx*n3 a >r > vawt- w * 

x^3tc t tyu^U1»S : E-Rc«jy§x.6h, HI 

1 (Cxv-Tct: die. If x*xV X/U-f 3 0 2 ©SffiTcfc 

v±i5fafcmMmMo<-y k-7 5 1 o^fflaa^jti 

£>„ BP5» ^-fVK'JS 1 0tt±#*>3 1 0c7)£Eptf 
sr*l«ili:Wi:*l«Dica»LT««*ti*. H12tcw 
r «fc o 04 y K-7 5 1 0 tt^a^SIB+OSffllcMf 

*Bttit**a5**-*. paitfiiivtLTs sia*. ft 

yx-f7a>x>y7'P/<-1'^1tt, HTML-¥>XML 



i 
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[0052] -mwmmojy k9#fj< t. ®9iz 
a** tiT^*T-^jwcai^Tiw** vic^r *w 

.IB#*V ±tf*V3"10 
T#*V3 12 

151** V 6***3 11 

feK*V3 13 
[0 0 5 3] 01 2lCSi1M:3lC, 10© 

3oo2tt«»ii5 1 ncjryHfcfiTjBy* 4§s© 

ifl5 1 2tt^T*r-fX^U'f 3 02©TfflT*«. C 
©4S@©SfcJtJ l Mgtt&l\ C©£fM85 1 
SJ#tt9 * > F9 5 1 0 tf KKMBAfMcfln LTJ: y 
£<©im fin-6, «ty*<«>y <f>*-5 

tt^fcfcSCckfc^LTVSo COttSZT, EH 3 ICS 
*\fcdlc» n— WSg±#^V3 1 0«ff?&. 01 
4tc^f«fc-5lCs 9*^95 1 OOSnHWCtfita 
L, Eft*1MUWai«A*. £©<!:*. *!lig#ffl->7. 
xA3ttU^/l/2»g ; E-RC«5o 01 5fcij**tlT 
t^cfedlC, 535.1 2£^t?9'f>F9 5 1 0©4O© 
ffl£-ncgJ?$S5 1 1 #anfctttt& 9-f V F9 5 1 

/V Ui*«Hrt\ 9-T> F95 1 OtCftfiLfcttHL 
[0 0 5 4] 01 6Tb^m2 0lt$.mm<J)O<<y\rO 

5 1 ofcics^nT^^v^-^x-r txu^v 

K0«!l*5?Lfct,O7?**. *T-f73>x>'.y9'PM 
<<*1 tOJ«S*?f3lcttSyy + >*iUt*y h9- 
^yah3^£H^ft«. *y h9-99P hu/Ui: 
LTBHTTP*y h7— ! ?9p h3iWWtU\ 0 
1 61*90 F9 5 1 0*£HlClfc&3li:«jjt$n« 

*>h©fliJ£l,T\ 20©**>5 13, 5 1 46^Sffi 

±tcss*tiTfcy, ctis©iaffiicas4-*i/c#*v 
&j y st 6tiT^*mm4t4i >t, wm$* 

*&K*>3 1 3tC»jyiT6*lTl^iSS«3t**V 

izakT&?tt$*>3 1 isjfrctKiy. 01 8 

iCSiT* 5 KHBKai** V 5 1 4 K A7D*t&££Mr 
*C 01 9Kjjvf<fc5K. «R#4«>3 1 

4£j?ti:> <>*-5?->a>£ltiTf3>;:<i:Jb ,t T- 

^vhsatarsc 020^* 

ic, 9-OK9 5 1 0©a>SH:CftKff-3T*«P*ti 



[0 0 5 5] »*7-rTi**jHRLn:t*KRllKl:-fV' 

•5»-5*->a>©aiitt> y*T-*<k'rv*-5*T 

*7xU* V h©7*o/tr*fcJ:ySfc»SftfcWH!:tt 

» jst « 9* > F9 tiusif -a-T «fc y ai^khm 

flHB*»*Cttf?*ntf, S*S<j«ll#*V3 1 0£ 
Jfrci:(cJ;y»S9'f>K9 5 1 OSCtcttCTtJ: 
9»CLTt»J:t\ W^'J^r-i/aVttHTML, J 
ava^'J^hTM, JavaTM, M^B/\-K 
9X79*5 v h7*-AlC«fcy^?K- h*tlTl**ffi© 

[ 0 0 5 6 ] 0 2 1 fcjj«r <fc 5 It, «WHftfT#* > 3 

1 25ff-T<!:. 9Y>F95 1 Otitis LTH;l&<fc 
y> «K#ffl->X7 1 A3tt02 2lcs-r<fe5lca>7 1 > 

kick*. 

[0 0 5 7] HXy>94Mi 
H6lC3iLfca>TVy««*-FflDtt«?» 02 31C 
5?T * => 'J h =1 > h □ -;WBB±fl!«Etf * > 3 1 

3*»rctK«feya«sttiBayv*^H*3HR« 

V*9'f> F9 5 2 Otf feTx*^ X9W 3 0 2©£ 

afreauusiers. nay 9-r v F9 5 2 0 tfH 
m$*y &WV3 13 

»Jt#4r> T**>3 12 

[0 0 5 8] 02 5tcsr«IT-tt, nm>jy^o-fyY 

9 5 2 OlCtt, "aa«3*4'tlTL'»*yT f -r 73>fV7 
fcH>ILfc1!S©y>?7V2j-5 2 1 ~5 2 5fl«« 
tlTl^, 02 5 ICS LfcftlTtt, "Japan T e 
am" ©7VA-5 2 1 tf A*»*i:«:-aT^*. 02 
6 K^iJrttTi^J: 3 (C^S^jMS?** > 3 1 4£fl 
LTy>*7>*-5 2 1tjWir*i:» 'JV^Jfelc^ 
ift-TSo y>*7>*-5 2 1 B9-M/ K9f K9X?* 
©^—>A©y V*T'*5©T% 9x9*9 f 59if79'y 

^g>ffi«±tfy» 02 7ic?s-r«fc5tcB*©-y 

•y»-^-/x©*-A^-5?5 2 ttW£t%&.oy*T 
t\ 02 8(C5St«fe3U:x 6#*>3 1 1*flift» 0 

2 9 icsr * 5 Knay> f 9-rv K9 5 2 o6^l:> 

* - V;U$iJ »» 9 X 9"9^ 9 -^-©tSTtcfeft^tl^o 

z. <»wm 9x995 9+f- k t p?^"o y m 

VhP-/l/gM3 0 9±© : E-K-y'f9;l/#*>3 1 5 
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*m t vxjy? o-v-*- Kfrstttts c t&?% 
s. zo&m. vm%m*sXTU3te®6izmLtc3> 

[0059] Hay > 29 -TV 5 2 OtfHftttlTl* 
SP& y*-h3Vhn-/l>gS3 0 9±4)Mii*'2> 

3 1 o<t^ji*2>3 1 2*fff cticiy, i«ay > 

^SfV K9 5 2 0rt<D'J>27>2j— £f60>O2- 
7^f-f7l^V K 5 2 1 ~5 2 5^tXr- r-TS 
C A^»a# ; E!2 5(C/Tx5'nT^ 

S«£?lC "Japan Team" <D7yl3—\z.$>%>t. 

03OlCf**<fc5fcTtf*V3 1 2*3^}frc:i: 
(Cfey, U3 1 tC^-Tcfc-plcA^^* "Record 
related e v e n t " <t£jj*#tlTVS'f V 
2 —5 2 r -f 7x V h 5 2 4 IC&ftfS c 

So 

[0060] c ovtmT\ m&mm 3 2 icat&fiTk* 

SP,8#2V3 1 3gL<ti3I*Rtf'2>3 1 4©ivfftfr 

sffi^T« 03 3ic^-r<fcaicBaay>2 , ?'f>F95 

2 0£MLS^©i|i*i£tfSC<htfT*;5 o EI3 41C 

it 5 2 4 a S 5 2 4 c $70 3 OOjURgBttib^So 
8tl8#W\ b5jS*^>3 1 0^&it*"2V3 1 2£tel^ 

a«#*>3 1 4T-nt7-rscticj:»; N ma^fx 

fcirr c t lz J: y fagfiiftfc** >-fe/UfS c 

So 

[0061] Hay>29-f>K2 5 2 0(ca^*ns 
utrsffl^^^sffl^oy >2« p-»;hbs3Vt 

>'^'J>*^*5. y>21ff$BttATVEFX 

Bars y >2 «£»s c s. 
[0062] ssamwBy? (ur d tjuvryv? 

2tt, xUcf^OUR I *ffl^TBS*ft*. xUtiB 

OUR I <DS**gfiE«s 

tv:<:broadcast>; 

735 y » :<:broadcast>;ttf ? — 9 V-T&fH LfcfcdDTfc 
y, C©x-2y-XttK*<VJ&->;*:rA (DNS) 

l/»S. 0J*.tfs 

tv:wqed. org for the WQED network 

tv:nbc. com for the NBC network 

[0 0 6 3] cn6ttaus©«ai?tt'f 

Tl>SIR?tt&<. MNRfcltiifTa h3/U*JHLT^ 



S„ -fey h h*7#yfrX3 2 0itl\,<y97<y-7f—7 
to*HtiLT&Js iV-y^T-yyT-yMZit-by h 
7#y *Z3 2 O#^?tttt£T0>tt£JR«)ttfil?0> 

Sr&Sft-CVSo ^JftT^SxUtfCDUR I h 
a;U©§!£ Ltd ?uettgafcft0£BfttiHN? 
tffcS. 

[0 0 6 4] ^tDxut'jaiM^ n^n^ttasQUB 

©1t«*Tl/tf0UR I KjD*SC£?-#ffl£*fcrr^ 
Sct5lc^S<!:^enT^So fljltf, 200 1^ 
D£#ffl->*xZ»3E1 0HO^8^ (P-*;U2f 
/*) It7*>y27.*«> h7-?£>3 0tt#lli§£®S-fS 

/c«>ictt s tv u r i«s:^o<fc^icffi3i-rs„ 

tv: fox. com/date=2001-0310/t ime=20:00:00/durat ion= 
0:30:00 

[0 0 6 5] a-W$fc»iS*nT^55:^Sffl / \(Dy 

05 P< .yfe-^i—tflC^^tlSo 
[0 0 6 6] 31?iS^5-tlT^S^T ; -r7=l>x> , yffl 
2 t— ? W > 2 - 5 9 7- -f ?'x U p« > h tf&l^i§ 

Hay>29'T> F95 2 otfa5*sti*u* 

A\ £SlHi7V7*>l/h©yy2£^A,f£IHIiiy>2 
9<VK9 5 2 0jb' t gjfv3-*lSo x-f 7*;Uh<0y>2 
t LTU\ /\- K9x7©HJfi7E©.'7i^—> r/ ^>y 

>2 v SI* 7-f 7©»^©9X7^- V"\<£> 'J 

« fclilE^i'NOy >9*^5tlSo 
[0 0 6 7] •/Xf^Ia-^i 

seoavf-v^K*- K^tc> a— »fttig3 sizm 

t cfc 5 IZ y h =1 V h □ -/USS 3 0 9 ©Ttf'$ > 3 
1 2^«i-rc<t:tc t i:oT->XxA^xa-»S^a^-r 
Silift^S. f-©^ 03 6lc^*tlT^S<J:5 
IC, ->Xx^p<Xa—?'r>K2 5 3 06^x*T'('X 
3 0 2<D±ffl#5T£|p|KS»S;i**tlSo ->X 
x^^-i-^-rv K9 5 3 0fl«Bllt>Tl>SH, ISS* 1 
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SYSTEM FOR INTERACTING WITH MEDIA CONTENT 
USING SLIDING GRAFICAL WINDOWS 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to media playback and 
Interactive television. 

2. Related Art 

A large number of interactive television systems and 
protocols have been developed in recent years. Interactive TV 
Bystents such as those based on the OpenTV 1 * specification 
already have millions of units in the . marketplace . With 
OpenTV™ , an interactive logo appears on the screen to 
indicate that the broadcast being viewed includes interactive 
content . When the logo is displayed, the viewer can use a 
remote control unit to display and scroll through interactive 
menus . 

Microsoft's WebTV™ enables live interaction during 
television broadcasts . and also offers access to email web 
content on television sets. Some types of interaction that is 
available includes playing along with game shows, 
participating in polls, and chatting with other viewers. An 
"i" icon is displayed on the television screen to indicate 
that a television broadcast includes interactive content. 

Other systems such as set top boxes made by TiVo™ and 
ReplayTV w< offer the ability to capture and store broadcast 
content on hard drives. 
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SUMMARY OF THE INVENTION 
The term "metadata" as used with respect to the 
present invention refers to information or data about media 
content. For example, information on a television program, 
such as the title of the program, is metadata about the 
program. A list of time stamps that mark where scenes begin 
in a video program is another example of metadata. Display 
of links to local or remote content, such as displayed web 
page URLs , are further examples of metadata. 

The term "interactive element" as used with respect to 
the present invention refers to an application involving 
human interaction through an interface device such as a 
remote control, keyboard, or mouse. Most interactive 
elements act on metadata, media data, or both. The most 
basic kind of interactive element is an application that 
simply enables the display of metadata, such as the display 
of program information. A more complex example of an 
interactive element that acts on metadata is an application 
for a poll, that enables a user to vote on some issue 
related to media content, wherein the issue of the poll and 
the candidate choices are metadata. An example of an 
interactive element that acts on media data is a link anchor 
to content such as a related television show, which is media 
content. The application that starts playback of the 
referenced show when the button is pushed is also an 
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interactive element. An example of on interactive element 
that acts on both media data and metadata is an interactive 
time bar that allows a user to navigate video by skipping 
forward to the next scene when a "next" button is pushed, 
and skipping backward to the previous scene when a 
"previous" button is pushed* The data that makes up the list 
of scene boundaries is metadata, and the video that it acts 
upon is media data. 

The term "semantic category" as used with respect to 
the present invention refers to classification of 
Interactive elements and pieces of metadata. According to 
the present invention, all pieces of metadata and interactive 
elements are assigned to semantic categories. For example, 
the system can include four semantic categories: information 
about the current program, system menu and setup, related 
links, and additional information. Each interactive element 
and piece of metadata relating to broadcast or stored media 
content is assigned to one, or sometimes more, of these 
semantic categories. 

One aspect of the present invention relates to a 
system for interacting with media content, metadata, and 
Interactive elements while the media content is output from 
a media output device. The media content can be visual 
content, audible content, or both.. Examples of media content 
include television programs, movies, sports shows, music. 
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radio shows* and the like. The media content can come from a 
variety of different sources. For example, the media content 
could be broadcast, stored locally, or distributed over the 
Internet. The media output device may include an audio 
amplifier and speaker, a visual display, or a combination of 
both. 

The system includes a media output device controller, 
assigning means, selection means, and output means. 

The media output device controller controls the media 
output device to output the media content. Examples of the 
media output device controller include an audio processor 
for controlling an audio amplifier and speaker to audibly 
output the media content, a graphics controller for 
controlling a display to visually display the media content, 
or a combination of both. 

The assigning means is used for assigning the metadata 
and the interactive elements to semantic ^categories . 

The selection means is use for selecting one of the 
semantic categories. The selection means can be any user 
interface, including a remote control, mouse, keyboard; 
joystick, or touch panel. 

The output means is used for output ting metadata and 
interactive elements that belong to the selected semantic 
category in a manner dependent on the selected semantic 
category. Examples of the output means Include an audio 
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processor for oon trolling an audio amplifier and speaker to 
audibly output the metadata and interactive elements, 
graphics controller for controlling a display to visually 
display the metadata and interactive element, or a 
combination of both. The output means and the media output, 
device controller can have shared or totally different 
configuration. 

With this configuration, when a user selects a semantic 
category, then all metadata and interactive elements assigned 
to the selected semantic category are outputted in a manner 
that corresponds to the selected semantic category. 

A device according to another aspect of the present 
invention Includes a display controller, assigning means, a 
selection unit, and a window generator. 

The display controller controls a display to display 
media content. The assigning means assigns the metadata and 
the interactive elements to semantic categories. The 
selection unit selects one of the semantic categories 
according to user input. 

The window generator generates a category window 
including metadata and interactive elements of the selected 
semantic category, and controls the display to superimpose 
the category window over the media content at a location 
that depends on the selected semantic category. 

For example, each of four semantic category described 
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above, that is, information about the current program, 
system menu and setup, related links, and additional 
information, can be associated with a different edge of a 
four-edged display screen. When input is received from a user, 
indicating selection of one of the semantic categories, for 
example when the user presses a corresponding arrow key of a 
remote control unit having four arrow keys, then a window 
including all of the metadata and interactive elements 
assigned to the selected semantic category will appear at the 
corresponding edge of a display screen. By pressing the key 
again, the window can further slide out when additional 
metadata or interactive elements are available for the 
selected category. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The above and other objects, features and advantages of 
the invention will become more apparent from, reading the 
following description of the embodiment taken in connection 
with the accompanying drawings in which i 

Fig. 1 is a block diagram showing entities involved in 
production, transmission, and reception of media content 
according to the embodiment of the present invention; 

Fig. 2 is a block diagram showing details of a 
broadcaster of Fig. 1; 

Fig. 3 (a) is a block diagram showing details of a 
viewer system of Fig. 1; 
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Fig. 3 (b) is a plan view showing details of a remote 
control shown in Fig, 3 (a); 

Fig. 4 is a flowchart representing processes performed 
to prepare and broadcast media content, metadata, and 
Interactive elements; 

Fig. 5 is a flowchart representing processes performed 
in the viewer system; 

Fig. 6 shows an example of media content displayed on 
a display of the viewer system; 

Fig. 7 is a table showing button id's assigned to 
physical buttons of the remote control of Fig. 3 (b); 

Fig. 8 is a table showing correspondence of logical 
and physical buttons depending on selected semantic 
category; 

Fig. 9 is a schematic view showing a button of the 
remote control being pressed for selecting a program details 
semantic category; 

Fig. 10 shows a program details window displayed on 
the display, upon selection of the program details semantic 
category; 

Fig. 11 is a magnified view of the program details 
window ; 

Fig. 12 is a schematic view showing a button of the 
remote control being pressed for further opening the program 
details window to full extension; 
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Fig, 13 shows the fully extended program details 
window displayed on the display ; 

Fig. 14 shows details of the fully extended program 
details window; 

Fig. 15 shows details of an interactive game displayed 
in the fully extended program details window; 

Fig. 16 is a schematic view showing a button of the 
remote control being pressed for shifting focus of input to 
another candidate in the Interactive game of Fig. 15; 

Fig. 17 shows focus of input in. the window being 
shifted to the candidate selected in the operation of Fig. 
16; 

Fig. 18 is a schematic view showing a button of the 
remote control being pressed for selecting the candidate 
that is the focus of input; 

Fig. 19 shows action performed when the candidate is 
selected; 

Fig. 20 (a) is a schematic view showing a button of 
the remote control being pressed for closing the program 
details window; 

Fig. 20 (b) shows the display screen after the program 
details window has been closed; 

Fig. 21 is a schematic view showing a button of the 
remote control being pressed for selecting a related links 
semantic category; 
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Fig. 22 Bhows a related links window displayed on the 
display when the related links semantic category Is 
selected; 

Fig. 23 Is an enlarged view showing the related links 
window Of Fig. 22; 

Fig. 24 Is a schematic view showing a button of the 
remote control being pressed for selecting a "Japan Team" 
anchor link in the related links window; 

Fig. 25 shows the display controlled to display the 
home page of the Japanese soccer team, by traversing the 
"Japan Team" anchor link according to the operation of Fig. 
24; 

Fig. 26 is a schematic view showing a button of the 
remote control being pressed for closing the related links 
window; 

Fig. 27 shows the display after the related links 
window Is closed, and control of the display is passed to a 
web browsing application; 

Fig. 28 is a schematic view showing a button of the 
remote control being pressed for browsing through Items in 
the related links window; 

Fig. 29 shows focus of input shifted to a "record 
related event" item after pressing the button indicated in 
Fig. 28; 

Fig. 30 is a schematic view showing a button of the 
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remote control being pressed for selecting the "record 
related event" item; 

Fig. 31 shows the related link window further opened 
to show items in the "record related event" item; 

Fig. 32 is an enlarged view of the related link window 
when opened as shown in Fig* 31; 

Fig. 33 is a schematic view showing a button of the 
remote control being pressed for selecting a system menu 
semantic category; 

Fig. 34 shows a system menu window displayed on the 
display when the system menu semantic category is selected; 

Fig. 35 is an enlarged view of the system menu window; 

Fig. 36 is a schematic view showing a button of the 
remote control being pressed for selecting an additional 
information semantic category; 

Fig. 37 shows an additional information window 
displayed on the display when the additional information 
semantic category is selected; 

Fig. 38 is an enlarged view showing the additional 
information window; 

Fig. 39 is a schematic view showing overall structure 
of an XML- encoded data stream used to associate a program id 
with metadata and interactive elements; 

Fig. 40 shows a concrete example of an XML-encoded 
data stream; 
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Fig. 41 shows relationship of major entities when an 
additional information provider is an independent 
organization. 

DETAILED DESCRIPTION OF THE EMBODIMENT 
A system according to the embodiment of the present 
invention will he described while referring to the 
accompanying drawings. 

First, an overall explanation of entities Involved 
with the present invention will be provided with reference 
to Fig. 1. A media content provider 1 provides media content, 
such as television programs, sports events, weather reports, 
and the like, to a broadcaster 2 over a media distribution 
channel lb. The broadcaster 2 distributes the media content 
to a viewer system 3 over a distribution channel 2a. 
Examples of broadcast distribution channels include 
terrestrial, satellite, cable distribution or other 
multiplexed television broadcast, in addition to 
distribution over the Internet. 

An additional information provider 4 provides 
additional information about the media content to the 
broadcaster 2 over a distribution channel 4a, for mixing 
with the media content by the broadcaster 2. Examples of 
additional information include metadata such as a list of 
characters on a TV program or advertisements, and 
interactive elements such as links to access information 
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about the characters on a TV program and links to web pages 
related to the TV program. In this embodiment, XML is used 
to represent metadata content, and Java™ programming 
language is used to represent interactive elements. A more 
detailed description of the metadata and interactive 
elements is discussed below. 

Fig. 2 shows an example of the broadcaster 2 based on 
the digital terrestrial television broadcasting model of the 
International Telecommunication Union. The system includes a 
video subsystem 210, an audio subsystem 220, a service 
multiplex and transport system 230, and an RF/ transmission 
system 240. 

The video subsystem 210 codes and compresses a video 
signal from the media content provider 1, using the MPEG- 2 
coding scheme set out in ISO/IEC IS 13818-1 and ISO/IEC IS 
13818-2, or using some other method. The audio subsystem 220 
similarly codes and compresses an audio signal from the 
media content provider 1, using the AC-3 audio coding 
standard, or using some other method. 

The service multiplex and transport system 230 
includes a service multiplexer 231 and a transport encoder 
232 for combining various components and preparing them for 
broadcast. The service multiplexer 231 multiplexes the 
compressed video and audio signals, and also ancillary data 
and control data, into one signal. The transport encoder 232 
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packetlzes the signal using the MPEG- 2 transport stream 
syntax. 

The RF/ transmission system 240 Includes a channel 
coder 241, a modulator 242, and a transmitter 243 for 
broadcasting the packetized signal from the service 
multiplex and transport system 230- The channel coder 241 
modifies the signal to protect against errors. The modulator 
242 modulates the digital signal over a carrier frequency, 
which is then broadcast using the transmitter 243. The 
viewer system 3 receives the signal over the channel 2a and 
reverses the encoding process to output digital audio and 
video signals. 

Fig. 3 (a) is a block diagram showing hardware for the 
viewer system 3 according to the embodiment of the invention. 
The viewer system 3 includes a set top box 320, a hard disk 
drive 301, a video display 302, a speaker 302a, and a simple 
remote control 309. 

The set top box 320 includes a processor 321, a MPEG-2 
decoder 322, a tuner 323, a graphics controller 324a, an 
audio processor 324b, an audio amplifier 324c, a hard disk 
controller 325, a network controller 326, a memory 327, and 
an infrared input/output (I/O) controller 328. The processor 
321 is connected to the MPEG-2 decoder 322, the tuner 323. 
and the graphics controller 324a by a graphics bus 329, and 
uses the graphics bus 329 to send commands and data to the 
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MFEG-2 decoder 322 , the television tuner 323. and the 
graphics controller 324a. The processor 321 is connected to 
the hard disk controller 325, the network controller 326, the 
memory 327 , the audio processor 324b, and the infrared 
input /output (I/O) controller 328 by a common bus 330, and 
uses the common bus 330 to read and write data, and to send 
control and status commands to the hard disk controller 325, 
the network controller 326, the memory 327, the audio 
processor 324b, and the infrared input/output (I/O) 
controller 328. 

The hard disk drive 301 is connected to the hard disk 
controller 325 by a bus 331. In this embodiment, the disk 
drive 301 is located physically external from the set top box 
320, but the disk drive 301 can be located physically Inside 
the set top box 320. The hard disk controller 325 controls 
the bus 331. The hard disk controller 325 can be an IDE or 
SCSI bus controller, for example. 

The network controller 326 1b connected to a network 
332, such as the Internet, and sends data to and receives 
data from the network 332. The network controller 326 also 
interprets control and status commands. The memory 327 is 
used by the processor 321 and other modules for the temporary 
storage of programs and data. The infrared I/O controller 328 
receives commands from the remote control 309, and either 
stores the commands in the memory 327 or sends the commands 
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directly to the processor 321. 

The audio processor 324b Is connected to the bus 331. 
The processor 321 sends control and status requests to the 
audio processor 324b, causing data and commands to be sent 
from the memory 327 , from the network controller 326, and/or 
from the hard disk controller 325 to the audio processor 324b. 
The audio processor 324b is additionally connected to the 
audio amplifier 324c by connection 333b. Those skilled in the 
art will recognize that the audio processor 324b and audio 
amplifier 333b may be eguivalently realized using a Bingle 
physical device such as a computer sound card that Includes 
both audio processor- and audio amplifier functionality. The 
audio amplifier 324c is connected to the speaker 302a by a 
cable 333c. The audio processor 324b controls the audio 
amplifier 324c and the speaker 302a to output sounds, such as 
radio broadcasts and the audio section of movies or 
television broadcasts. 

The tuner 323 demodulates television signal from the 
broadcaster 2 sent over the channel 2a. The graphics 
controller 324a processes graphics commands and performs 
functions such as video compositing, drawing, and scan line 
conversion. The graphics controller 324a is connected to the 
video display 302 by a video cable 333. The audio processor 
324b is connected to the audio amplifier 324o by a cable 333b, 
and the audio amplifier 324a is connected to the speaker 302a 
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by a dable 333c. The audio processor 324b controls the audio 
amplifier 324c and the speaker 302a to output sounds, such as 
radio broadcasts, and the audio section of movies or 
television broadcasts. The MPEG-2 decoder 322 decodes 
compressed video data from the hard disk 301 and the tuner 
323. 

Fig. 3 (b) is a plan view showing details of the remote 
control 309. The remote control 309 includes six navigational 
buttons, that is, an up button 310, a right button 311, a 
down button 312, a left button 313, a select button 314, and 
a mode cycle button 315. . 

Next, operations of the system according to the 
present embodiment will be described while referring to the 
flowcharts in Figs. 4 and 5. The flowchart of Fig. 4 
represents processes performed by the media content provider 
4, the broadcaster 2, and the additional information 
provider 4. The flowchart of Fig. 5 represents processes 
performed by the viewer system 3. 

First in S300, media content Including video and audio 
content is produced by the media content provider 4. Also, a 
unique id is assigned as ancillary data for the media 
content. Assigning a unique id, that is, program id, is a 
well known process, and is required for digital broadcasts. 
For example, the Advanced Television Systems Committee 
(ATSC) describes standards for program identifiers in its 
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standard document A/57* 

In S301 the additional information provider 4 creates 
metadata and interactive elements associated with the media 
data to be broadcast as ancillary data. Digital broadcast 
standards already define a standard way for representing 
some kinds of metadata. For example the Advanced Television 
Systems Committee (ATSC) describes standards for encoding 
program and system information for terrestrial and cable 
broadcast in its standard document A/65. Some standards for 
interactive elements used with broadcasts are already in 
widespread use. For example , the Advanced Television 
Enhancement Forum (ATVEF) defines a standard that is based 
on HTML 4.0 f ECMAScript and DOM. Other standards, such as 
DVB-MHP and ATSC-DASE , are based on Java™ programming 
language, but are still being finalized. 

Also in S301, the additional information provider 4 
associates the metadata and interactive elements data with 
the program id of the program to which the metadata and 
interactive elements relate, so that correlation with the 
correct program can be made when the data is reconstructed 
in the viewer system 3. Further in S301, the additional 
information provider 4 associates each piece of metadata and 
interactive elements with one, and sometimes more, semantic 
categories. The semantic category of a piece of metadata or 
interactive element determines the manner in which that 
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metadata or interactive element will ultimately be displayed 
by the viewer system 3. A detailed example of how metadata 
and interactive elements can be associated with semantic 
categories is explained later. 

Next, in S302 the broadcaster 2 multiplexes the media 
data with the ancillary data, that is, with the program id, 
metadata, and interactive elements from S300 and S301. The 
broadcaster 2 digitally broadcasts using MPEG- 2 as a 
transport stream. The service multiplexer and Transport 
System 230 performs this multiplexing operation using the 
international standard for Digital Storage Media Command and 
Control (DSM-CC) defined in ISO/IEC 13818-6, and more 
specifically the data carousel protocol. 

In S303, the modulator 242 modulates the multiplexed 
signal with other channels and the RF/transmission system 
243 broadcasts the signal using a standard digital broadcast 
protocol such as ATSC or DVB. 

Turning to Fig. 5, in S304 the viewer system 3 
receives the modulated signal from the broadcaster 2. The 
tuner 323 tunes the signal, that is, demodulates the signal,, 
to a single channel, and extracts a digital signal for one 
channel from the demodulated signal. This digital signal 
still contains both media data, metadata, and interactive 
elements. 

In S305, the MPEG-2 decoder 322 digitally 
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demultiplexes the tuned signal using existing digital TV 
application programming interfaces. During the 
demultiplexing operation, the media data is separated from 
the metadata and interactive elements to produce a media 
content stream and a data content stream, wherein the data 
content stream contains the metadata and interactive 
elements . 

In S306, the processor 321 performs control processes 
to store each piece of metadata or interactive element in 
the data content stream as a table in the memory 327, or as 
part of a local database in the HDD 301. It should be noted 
that the program id associated with the metadata and 
interactive elements in S301 is used as a key to access the 
data in S3 10 below. 

In S307, the processor 321 performs control processes 
to extract the semantic categories from the metadata and the 
interactive elements, and store the semantic categories in 
correspondence with the metadata and the interactive 
elements. It should be noted that the semantic categories 
are used as secondary keys to search and retrieve 
corresponding metadata and interactive elements in S310 
below. 

In S308, the graphics controller 324 and the processor 
321 operate to display the media content on the display 302 
based on the media content stream from S305. An example of 
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media content displayed on the display 302 in the content 
viewing mode i8 the soccer game 500 shown in Fig. 6. The end 
user views the media content on the display 302. Viewing the 
media content is the primary mode for a viewer to interact 
with the system, and will be referred to as the content 
viewing mode hereinafter. 

At this time, an indication is provided if metadata, 
interactive elements, or both are available. An example of 
such an Indication is a graphical indication superimposed on 
the display of the media content. Configuration required for 
implementing such a graphical indication is disclosed in U.S. 
Patent No. 5, 848. 352- to Dougherty, et al. The disclosure of 
U.S. Patent No. 5,848,352 is hereby incorporated by 
reference . 

In S309, input for selecting one of the four semantic 
categories is received from the remote control 309. That is, 
to access metadata and interactive elements of various 
semantic categories, the viewer must switch the system from 
the content viewing mode to a level 1 semantic category mode. 
The viewer can do this by using the remote control 309 to 
select the semantic category that corresponds to the desired 
metadata and interactive elements. In the embodiment, there 
are four semantic categories: program details, related links, 
system menu, and additional information. It should be noted 
that not all of the semantic categories are assigned 



20 



(39) 



ftH 2001-292425 



metadata and interactive elements associated with the 
presently displayed media content. For example, the system 
menu category is assigned metadata and interactive elements 
that is unrelated with the media content. 

While in the content viewing mode, each of the buttons 
110 to 113 on the remote control 309 is assigned one 
semantic category. That is, the button 310 is assigned the 
detail on current program category, the button 313 is 
assigned the related links category, the button 311 is 
assigned the additional information category, and the button 
312 is assigned the system menu category. The viewer can 
switch the system from the content viewing mode to a level 1 
category mode by pressing the button that corresponds to the 
semantic category of the desired metadata and interactive 
elements. 

In S310, if the selected semantic category is an 
active category and the selected semantic category is one 
related to the presently display media content, then the 
system switches from the content viewing mode to the level 1 
category mode, and metadata and interactive elements that 
match both the current program id and the selected semantic 
category are retrieved. The set top box 320 knows the 
program id of the media content that Is currently being 
viewed, because the program id is embedded in the media 
content. If the selected semantic category is one such as 
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the system menu, which includes only metadata and 
interactive elements that are unrelated to the presently 
displayed media content, then only metadata and interactive 
elements that match the selected semantic category are 
retrieved. 

The retrieved metadata and interactive elements are 
parsed and displayed on the display 302 in a sliding window 
that corresponds to the selected semantic category. When the 
selected semantic category is one that Includes only 
metadata and interactive elements that are unrelated to the 
presently displayed media content, then the corresponding 
window will remain the same regardless of the displayed 
media content. 

Also in S310, the system redefines the function of the 
physical buttons 110 to 113. That is, four logical buttons, 
open, close, forward, and backward are defined to map to 
different physical buttons 110 to 113 depending on which 
semantic category is selected. Each of the physical buttons 
110 to 113 is represented by a button id as shown in Fig. 7. 
The correspondence between logical buttons and physical 
buttons is set depending on the currently chosen semantic 
category as shown in Fig. 8. 

In final step S311, the user interacts with the 
metadata or interactive element in a manner that depends on 
the particular interactive element. 
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Here, a detailed example of operations performed in 
S308 to S311 as seen by the viewer will be described. As 
mentioned previously, in the embodiment each semantic 
category is associated with one of the four edges of the 
display 302* When a semantic category is selected, then a 
window with the metadata and interactive elements of that 
category will slide out from the corresponding edge of the 
display 302 # following the direction indicated by the arrow 
of the button on the remote control 309. 

Program Details Category 

While the viewer is watching the soccer game 500 in 
the content viewing . mode shown in Pig . 6 , if the viewer 
presses the up button 313 as shown in Fig. 9, then the 
viewer system 3 switches into the level 1 category mode and 
as shown in Fig. 10 a program details window 510 slides up 
from the bottom edge of the screen on the display 302. That 
is. the window 510 slides in the same direction as indicated 
by the arrow of the up button 310. As shown in Fig. 11, the 
window 510 shows details about the program currently being 
viewed. Such detail Includes program name, channel, date and 
time the program was broadcast, and program length. The 
media content provider 210 can specify other program details 
using a standard markup language such as HTML or XML. 

Once the program detail window is open, the following 
assignments are made for logical buttons based on the table 



23 



(42) 



1^2001-292425 



Id Fig. 8. 



Logical Button 



Physical Button 



OPEN BUTTON: 



UP BUTTON 310 



CLOSE BUTTON i 



DOWN BUTTON 312 



FORWARD BUTTONS 



RIGHT BUTTON 311 



BACKWARD BUTTON i 



LEFT BUTTON 313 



As shown in Fig. 11, a border 511 surrounds the window 
510 on three sides only. A fourth side 512 of the window 510 
abuts the edge of the display 302, without the border 511. 
This missing section of border 511 indicates to the viewer 
that the window 510 can be further slid out to reveal more 
information, that is, more metadata, interactive elements, 
or both. In this situation, when the user again presses the 
button 310 as shown in Fig. 12, then the window 510 opens 
further as shown in Fig. 13 to display an additional level 
of information. At this time, the viewer system 3 is in a 
level 2 category mode. As shown in Fig. 14, the border 511 
extends on all four sides of the window 510, Including the 
side 512. This Indicates that the window is fully extended. 
No further metadata or interactive elements exist for this 
semantic category, and no further extension of the window 
510 is possible. 

Figs. 15 to 19 depict an example of an interactive 
element in the fully-extended window 510. Interactivity with 
the media content provider 210 requires a back channel and a 
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network protocol, preferably the HTTP network protocol. Fig. 
15 shows the window 510 in the initial configuration, just 
after the window 510 is fully extended. The example 
interactive element is represented by two on-screen buttons 
513 and 514 that enable the user to participate in a poll 
about the soccer game. The logical forward, backward and 
select buttons are available for use by interactive elements. - 
That is, the viewer uses the forward logical button, which 
is assigned to the physical button 311 at this time, and the 
backward logical button, which is assigned to the physical 
button 313 at this time, to change the focus of button input. 
Thus, pressing the button 311 as shown in Fig. 16 changes 
focus of button input to the on-screen button 514 as shown 
in Fig. 17. Pushing the select button 314 as shown in Fig. 
18 completes the interaction and sends an action event to an 
Interactive application. The display of window 510 is 
controlled accordingly as shown in Fig. 19. 

What kind of Interaction will occur when an item is 
selected will depend on the design chosen by the provider of 
the metadata and the interactive elements. For example, the 
system can be designed so that when an item is selected, 
then the corresponding window will slide open more, and the 
corresponding interactive application in the window will be 
executed. If additional information is made available as a 
result of the interactive operation, then the category 
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window 510 will open further when the viewer presses the 
open button 310. The application can be implemented in HTML, 
JavaScript™, Java™ or other language as supported by the 
hardware platform. 

When the user presses the button 312 as shown in Fig. 
20 (a), then the window 510 slides out of sight, and the 
viewer system 3 returns to the content viewing mode as shown 
in Fig. 20 (b). 

Related Links Category 

While in the content viewing mode of Fig. 6, then the 
viewer can select the related links category by pressing the 
left remote control button 313 as shown in Fig. 21. As a 
result, as shown in Fig. 22 a related links window 520 
scrolls out from the right edge of the display 302. While 
the related links window is open, logical buttons are 
assigned to the physical buttons as follows: 

Logical Button Physical Button 

OPEN BUTTON t LEFT BUTTON 313 

CJ.OSE BUTTON i RIGHT BUTTON 311 

FORWARD BUTTON: DOWN BUTTON 312 

BACKWARD BUTTON* UP BUTTON 310 

In the example shown in Fig. 23, the related links 
window 520 includes a plurality of link anchors 521 to 525 
related to the present media content. In the example of Fig. 
23, Japan Team anchor 521 has the input focus. When the 
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viewer eeleots the link anchor 521 by pressing select button 
314 as shown In Fig. 24, the link Is traversed. Because the 
link anchor 521 is a link to a world wide web page, a web 
browser application is opened and the display 302 is 
controlled accordingly to display the home page 526 of the 
Japanese soccer team as shown In Fig. 25. If at this time 
the user presses the button 311 as shown in Fig. 26, the 
window 520 closes as shown in Fig. 27 and cursor control is 
passed to the web browser. This is called the WEB browser 
mode. The viewer can leave the WEB browser mode by pressing 
the mode cycle button 315 on the remote control 309. As a 
result, the system 3. returns to the content viewing mode of 
Fig. 6. 

While the window 520 is open, the viewer can navigate 
among the link anchors and other Interactive elements 521 to 
525 of the window 520 by pressing the forward and backward 
buttons 310 and 312 on the remote control 309. For example, 
while the focus of input is on the Japan team anchor 521 as 
shown in Fig. 23, then by pressing the down button 312 as 
shown in Fig. 28 three times, the focus of input can be 
shifted as shown in Fig- 29 to the interactive element 524 
labeled record related event. 

In this condition, the viewer can use either the open 
button 313 as shown in Fig. 30 or the select button 314 to 
further open the window 520 as shown in Fig. 31 to access 
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more choices. As shown in Fig. 32, the fuliy opened window 
520 has three selections 524a to 524c. The viewer navigates 
to the desired item using the forward and backward buttons 
310 and 312, and selects using the select button 314. The 
viewer may instead cancel the record action by pressing the 
close button 311. 

Examples of links displayed in window 520 include 
links to web pages, to other television or radio programs, 
or to locally stored content. Link information is preferably 
delivered with media content using the ATVBF enhanced 
television specification, or alternately over the Internet 
or other network. The additional information provider 4 can 
optionally include links about a program when the program is 
distributed. 

Links are specified using Uniform Resource Identifiers 
(URI). For example, links to TV content are specified using 
URI* a for television broadcasts. The basic structure of a 
television URI is: 

tvt < broadcast > 

Where <broadcast> is a description of the data source. 
This data source can use an identifier such as used in the 
domain name system (DNS), The identifier indicates a 
particular broadcaster or television network. For example: 

tv: wqed.org for the WQED network 

tvmbc.com for the NBC network 
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Note that these broadcaster identifiers are not the 
same identifiers as Internet addresses, and so should not 
resolve to IP addresses. Instead, they must resolve to 
frequencies and broadcast protocols. The set top box 320 
includes a lookup table that contains a list of all 
broadcast identifiers that the set top box 320 can tune in 
to, and the corresponding frequency and protocol. An example 
of TV URI protocol being developed is the Uniform Resource 
Identifiers for Television Broadcasts. 

Future television broadcasts can be referenced by 
adding starting time and broadcast duration information to 
the TV URI. For example, to record a 30 minute program on 
the Fox network on March 10. 2001 at 8:00pm (local time), a 
tv url scheme could be extended as follows t 

tv: f ox.com/date»2001-0310/ time =20 : 00 * 00/duration=0 x 30 1 00 

When the user follows a link to a program that has not 
yet aired, the user can be shown a message explaining that 
the program has not yet aired. 

If no metadata or interactive elements are available 
for the currently displayed media content, then either the 
related links window 520 is not displayed, or the related 
link window 520 can be displayed with default links, such as 
links to the web page of the manufacturer of the hardware, 
links to the web page for the broadcaster of the current 
media, and links to advertisers. 
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System Menu Category 

During the content viewing mode of Fig. 6, the user 
can select the system menu category by pressing the down 
button 312 of the remote control 309 as shown in Fig. 33. As 
a result, a system menu window 530 scrolls down from the top 
edge of the display 302 as shown in Fig. 34. While the 
system menu window 530 is open, logical buttons are assigned 
to physical buttons as follows: 

Logical button Physical Button 

OPEN BUTTON! DOWN BUTTON 312 

CLOSE BUTTON r UP BUTTON 310 

FORWARD BUTTON:- RIGHT BUTTON 311 

BACKWARD BUTTON: LEFT BUTTON 313 

The system menu window 530 gives access to system 
settings such as user preferences, allows users to launch 
applications, and allows the user to browse locally stored 
content. In the example of Fig. 35, the system menu window 
530 includes choices 531 to 535. As in the previously 
examples, the viewer uses forward and backward buttons to 
navigate through the list. 

Additional Information Category 

During the content viewing mode of Fig. 6, the user 
can select the additional information category by pressing 
the button 311 as shown in Fig. 36. As a result, an 
additional information window 540 slides out from the left 
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edge of the display 302 as shown In Fig. 37. While the 
additional window Is open, logical buttons are assigned to 
physical buttons as follows: 

Logical Button Physical Button 

OPEN BUTTON t RIGHT BUTTON 311 

CLOSE BUTTON: LEFT BUTTON 313 

FORWARD BUTTONS DOWN BUTTON 312 

BACKWARD BUTTON: UP BUTTON 310 

In the example shown in Fig. 38, the additional 
Information window 540 contains selectable Items 541 to 546. 
The viewer can navigate through the list using the forward 
and backward logical* buttons, and can select an item using 
the select button 314. 

Here, the process in S301 for associating a piece of 
metadata or an interactive element with a semantic category 
will be explained in further detail. Each piece of metadata 
and interactive element includes a tag that correlates the 
piece of metadata or the interactive element with one of the 
semantic categories. Such a tag can be realized by encoding 
and distributing the metadata and interactive elements as an 
extensible Markup Language (XML) document. XML is well known 
and Is defined in specifications recommended by the World 
Wide Web Consortium (W3C). 

Fig. 39 schematically shows the overall structure of 
an XML-encoded data stream 406 used to associate a program 
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Id with metadata and interactive elements. The XML-encoded 
data stream 406 also associates a semantic category with 
metadata and interactive elements. The data stream 406 can 
also contain the metadata and interactive elements 
themselves. 

The XML-encoded data stream 400 includes fields 401 to 
405. Field 401 contains a reference to the program id. 
Fields 402 to 405 contain the metadata and interactive 
elements for each semantic category that the system defines. 
In the example, four semantic categories have been defined 
and assigned to the integers 1 to 4, which are the same as 
the above- described > four semantic categories of program 
details, related links, system menu, and additional 
information. 

The XML-encoded data stream 406 includes a link to a 
cascading style sheet 400, which is stored in both the 
broadcaster 2 and the viewer system 3. The casaadlng style 
sheet 400 contains rules describing how the metadata and 
Interactive elements should be displayed. For example, the 
cascading style sheet 400 allows for the specification of 
fonts, font sizes, color, and other attributes. Cascading 
Style Sheets are well known and are defined in 
specifications recommended by the World Wide Web Consortium 
(W3C) . 

Fig. 40 shows further detail of the XML-encoded data 
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stream 406. Line 00 describes a link to the cascading style 
sheet named • default .ess" . The style sheet may be stored 
locally, or it may have been broadcast via a data carousel 
(DSM-CC) as described earlier. Formats for linking a 
cascading style sheet to an XML document are well known. 

Line 02 contains the program Identifier which is used 
to correlate all information in the XML stream to a 
broadcast program. 

Lines 04 and 05 contain data for semantic category 1. 
In the example, semantic category 1 contains no metadata or 
interactive elements. 

Lines 07 to 12 contain the data for semantic category 

2. Lines 08 and 09 are links to web content. 

Lines 14 to 21 contain the data for semantic category 

3. The title attribute, which indicates the title of the 
television program, is assigned in line 15. Other attributes 
of the television program are assigned in lines 16 to 18. 
Lines 19 to 20 show one way that an Interactive element can 
be defined. The APPLET tag indicates that a Java™ applet 
should be included in that semantic category. In the example, 
the applet's code (that is, compiled byte-code) is 
referenced as: 

Iidi//12abc554o3d3dd3fl2abc554o3d3dd3f/code/timebar. jar 
This string is a local identifier that specifies where 
to find a Java™ archive file in the local file system that 
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Implements the Interactive element. A Java™ archive file, 
also called jar file, is a single file that bundles several 
Java™ class files and associated information files into a 
single archive file. A JAR file normally contains the class 
files and auxiliary resources associated with applets and 
applications. The lid local identifier format is well-known 
and defined by standards introduced by the Advanced 
Television Enhancement Forum (ATVEF) . 

When the user selects an interactive element 
referenced by a jar archive, a Java™ virtual machine opens 
and verifies the contents of the jar archive. The manifest 
file in the jar archive specifies which Java™ class file 
should be executed. The Java™ class file to be executed is 
specified as the Main-Class attribute. The Java™ execution 
engine calls the main function of the specified Java™ class 
file. Execution of Java™ classes, including those contained 
in jar files is well known in the art. 

While the invention has been described in detail with 
reference to specific embodiments thereof, it would be 
apparent to those skilled in the art that various changes and 
modifications may be made therein without departing from the 
spirit of the invention, the scope of which is defined by the 
attached claims. 

For example, in the embodiment the media content is 
broadcasted. However, the media content could be stored 
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locally, or distributed via the Internet. 

In the embodiment, the media content provider 1, the 
broadcaster 2, and the additional information provider 3 are 
all independent entitles. However, these can all be the same 
entity. Also, the functions performed by the media content 
provider 1, the broadcaster 2, and the additional 
information provider 3 can be realized by any number and 
combination of different entitles. For example, the media 
content provider 1 and broadcaster 2 can be the same entity, 
and the additional information provider can be a separate 
entity. 

In the embodiment, the additional information provider 
4 provides the additional information at the same time and 
using the same distribution channel as the media content, by 
embedding the additional information into the same MPEG- 2 
stream as the media content. However, as shown in Fig. 41 an 
additional Information provider 4' can distribute the 
additional information over a channel 4a' that is different 
from how the media content is distributed. For example, if 
the media content is a broadcasted TV program, then a third- 
party "TV Program Commentary" service might distribute 
additional information about content of the TV program over 
the Internet after the TV program has already been broadcast 
Such a scenario is particularly relevant to media content 
that has been stored on a local storage device of the viewer 
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system 3. 

Also, for simplicity the embodiment describes metadata 
and interactive elements as being assigned to entire 
programs. However, the present invention also applies to 
cases where metadata and interaotive elements are associated 
with only a portion of a program. In such cases, the 
metadata and interaotive elements must also contain 
information about the start time and duration of the program 
segment to which the metadata and interactive elements apply. 

Also, in the embodiment, the remote control 309 is 
given as an example of an interface that allows a user to 
select a 1 semantic category. However, other human/system 
interfaces, such as a keyboard or a mouse, can be used 
instead. 

Although interactive elements are represented by 
Java™ programming language in the embodiment, they can be 
specified using any interaotive language such as 
JavaScript™. 

Although the embodiment described the present invention 
applied to media content that is output both visually and 
audibly, the present invention could also be applied to media 
content that is only visually or only audibly output. 

Although the embodiment described displaying metadata 
and Interactive elements of different semantic categories at 
different edges of the display, the metadata and interaotive 
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elements of different semantic categories can be outputted In 
any way that enables a user to easily distinguish between the 
different semantic categories. For example, metadata and 
interactive elements of different semantic categories can be 
displayed using different colors , or output using different 
sounds, such as different voices. 

It should be noted that S306 and S307 can be combined 
into a single step. 

Also, the media content can be provided with a 
hlerarohal structure, and the remote control can be used to 
navigate through the hierarchical structure of the media 
content. A simple example of hierarchical structure provided 
to the media content is merely the order in which a program 
is normally shown. In this case, the remote control can be 
used to fast forward or fast reverse the program, or to skip 
to different scenes of the program. A more complicated 
example of hierarchical structure provided to the media 
content is branches in games or dramas, that allow a viewer 
to enjoy a variety of different outcomes for the same story. 
In this case, the remote control can be used to choose which 
one of the branches to follow. 
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What is claimed 1st 

1. A system for Interacting with media content, 
metadata, and interactive elements while the media content 
is output from a media output device, the system comprising i 

a media output device controller that controls the 
media output device to output the media content; 

assigning means for assigning the metadata and the 
interactive elements to semantic categories; 

selection means for selecting one of the semantic 
categories ; and 

output means for outputting metadata and interactive 
elements that belong. to the selected semantic category in a 
manner dependent on the selected semantic category- 

2. A system as claimed in claim 1, further comprising 
a storage device for storing the media content. 

3. A system as claimed in claim 1. further comprising: 
a television receiver for receiving a television 

signal from a broadcaster, the television signal including 
the media content; and 

a media content extractor that extracts the media 
content from the television signal. 

4. A system as claimed in claim 3, wherein the 
television signal further includes the metadata and the 
interactive elements, the media content extractor further 
extracting the metadata and the Interactive elements from 
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the television signal. 

5. A system as claimed in claim 3, further comprising 
a metadata/interactive element receiver that receives the 
metadata and the interactive elements from a third party not 
affiliated with the broadcaster of the television signal. 

6. A system as claimed in claim 3, wherein the media 
output device controller is adapted for controlling a 
television display to display the media content. 

7. A system as in any of claims 1-6, wherein the 
media output device controller is adapted for outputting at 
most one semantic category at one time. 

8. A system as in any of claims 1-7, wherein the 
media contents, the metadata, and the interactive elements 
have a hierarchical structure of at least one level, and 
further comprising a navigation means for navigating the 
hierarchical structure. 

9. A system as in any of claims 1-8, wherein said 
selection means includes a set of buttons, each button 
corresponding to one semantic category. 

10. A system as claimed in claim 9, further comprising 
a remote control unit with physical buttons that serve as 
the buttons of the selection means. 

11. A system as in any of claims 1-10, wherein the 
media output device is a display that displays the media 
content, and the output means controls the display to output 
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metadata and interactive elements of each selected semantic 
category in a separate category window superimposed over the 
media content. 

12. A system as claimed in claim 11, wherein the 
output means includes a hierarchy navigation means for 
navigating the selected semantic category's hierarchical 
structure. 

13. A system as claimed in claim 12, wherein said 
hierarchy navigation means includes a set of buttons. 

14. A system as in claim 11, wherein the output means 
controls the display to display each category window 
associated with and immediately adjacent to a different edge 
of the display. 

15. A system as claimed in claim 14 wherein the output 
means controls the display to display the category windows 
using a sliding effect so that the category window of the 
selected semantic category appears to slide open from the 
immediately adjacent edge toward an immediately opposite 
edge • 

16. A system as claimed In claim 15, wherein the 
output means includes a hierarchy navigation means for 
navigating the selected semantic category's hierarchical 
structure, the hierarchy navigation means controlling the 
display to further slide open the category window of the 
selected semantic category in association with navigation 
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down the hierarchy. 

17. A system as claimed In claim 16 , wherein the 
output means controls the display to display the category 
window of the selected semantic category with a visual cue 
to indicate that further metadata, interactive elements, or 
both relating to the selected semantic category is available. 

18. A system as claimed in claim 17, wherein the 
output means controls the display to display the category 
window of the selected semantic category surrounded by a 
complete encompassing border when the category window is 
fully extended, and with an incomplete border along the 
corresponding edge of the display when further metadata, 
interactive elements, or both relating to the selected 
semantic category is available. 

19. A system as in any of claims 1-18, wherein one of 
the semantic categories Includes an index for a plurality of 
media contents, and further comprising means for browsing 
the index. 

20. A system as in any of claims 1-19 wherein at 
least one of the semantic categories Includes metadata, 
interactive elements, or both for launching applications, 
the metadata, interactive elements, or both not changing 
regardless of the media content currently being output ted. 

21. A sys tern as in any of claims 1-20, wherein at 
least one of the semantic categories includes a list of 
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links to content related to the media content currently 
being output ted. 

22. A system as in any of claims 1-21, further 
comprising a means for navigating within the media content 
presently being output* 

23. A system as in any of claims 1-22, wherein at 
least one of the semantic categories includes additional 
information about the content currently being outputted. 

24. A system as in any of claims 1-23, further 
comprising indication means for indicating to a viewer that 
metadata, interactive elements, or both is available in one 
or more semantic categories. 

25. A system as claimed in claim 24, wherein the media 
output device includes a display that displays the media 
content, and the indication means includes a graphical 
Indication superimposed on the media content on the display. 

26. A system as in any of claims 1-25, further 
comprising network means for connecting with a network. 

27. A system as claimed in claim 26, wherein 
additional information about content is communicated over 
said network means. 

28. A system as claimed in claim 27. wherein media 
content is delivered over said network means. 

29. A system as claimed in claim 1, wherein the media 
output device includes a display, and the media output 
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device controller controls the display to visually display 
the media content. 

30. A system as claimed in claim 1, wherein the media 
output device includes a display, and the output means 
controls the display to display metadata and interactive 
elements of the selected semantic category. 

31. A system as claimed in claim 1, wherein the media 
output device Includes an audio amplifier and speaker, and 
the media output device controller includes an audio 
processor for controlling the audio amplifier and speaker to 
audibly output the media content. 

32. A system .in claim 1 wherein the output means 
includes an audio amplifier and speaker that audibly outputs 
metadata and interactive elements of the selected semantic 
category. 

33. A method for interacting with media content, 
metadata, and interactive elements while the media content 
is output from a media output device, which comprises: 

controlling the media output device to output the 
media content; 

assigning the metadata and the interactive elements to 
semantic categories; 

selecting one of the semantic categories; and 

outputting metadata and interactive elements that 
belong to the selected semantic category in a manner 
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dependent on the selected semantic category. 

34. A device for displaying visual content, metadata, 
and interactive elements on a display, the device 
comprising; 

a display controller that controls the display to 
display the visual content; 

assigning means for assigning the metadata and the 
interactive elements to semantic categories; 

a selection unit that selects one of the semantic 
categories according to user input; and 

a window generator for generating a category window 
including metadata and interactive elements of the selected 
semantic category, and controlling the display to 
superimpose the category window over the visual content at a 
location that depends on the selected semantic category. 
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<?xml-styleflheet href-"default .csa" type-"text/csa"?> 

< PROG RAM ID-' r 184932493' r > 

<METADATA 5EMANT I C_CATEGORY= "1 "> 
</METADATA> 

< METADATA SEMAHTIC_CATEGORY- l '2'*> 

<A HREF= w http:/7www.fox.com'*>FOX home page</A> 
<A HREr= , 'http://www.theai«ipson9.cooca">The Simpsons home 
page</A> - 

< /METADAT A> 

<METADATA SEMANTIC_CATEGORY-*3"> 
<TITLE>The Simpaona</TITLE> 
<SDBTITLE>Saddleaore Galatica</SUBTITLE> 

<BROADCASTER>B'OX</BROADCASTER> 
<AIRTIME>20:00 20:30</AIRTIME> 

<APPLET C odebaBe= w lid://12abc554c3d3dd3fl2abc554c3d3dd3f/ 
code / t imeba r . j ar" >< /appl et > 

< /MET ADAT A> 

<METADATA SEMANTIC CATEGORY-" 4 "> 
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ABSTRACT 

A system is proposed for displaying and interacting 
with media content. Metadata and interactive elements 
related to the media content are allocated to different 
categories. When a category is selected, then the metadata 
and interactive elements of the selected category are 
displayed on a television screen in a window that 
corresponds to the selected category. Each window elides out 
from a corresponding edge of the screen. When additional 
information is available related to a selected category or 
interactive element, the window slides out further when 
directed by the user.. 
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